<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: OpenCV iOS - Video Processing</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d9/df8/tutorial_root.html">OpenCV Tutorials</a></li><li class="navelem"><a class="el" href="../../d3/dc9/tutorial_table_of_content_ios.html">OpenCV iOS</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">OpenCV iOS - Video Processing </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p><b>Prev Tutorial:</b> <a class="el" href="../../d3/def/tutorial_image_manipulation.html">OpenCV iOS - Image Processing</a></p>
<table class="doxtable">
<tr>
<th align="right"></th><th align="left"></th></tr>
<tr>
<td align="right">Original author </td><td align="left">Eduard Feicho </td></tr>
<tr>
<td align="right">Compatibility </td><td align="left">OpenCV &gt;= 3.0 </td></tr>
</table>
<p>This tutorial explains how to process video frames using the iPhone's camera and OpenCV.</p>
<h2>Prerequisites: </h2>
<ul>
<li>Xcode 4.3 or higher</li>
<li>Basic knowledge of iOS programming (Objective-C, Interface Builder)</li>
</ul>
<h2>Including OpenCV library in your iOS project </h2>
<p>The OpenCV library comes as a so-called framework, which you can directly drag-and-drop into your XCode project. Download the latest binary from <a href="http://sourceforge.net/projects/opencvlibrary/files/opencv-ios/">http://sourceforge.net/projects/opencvlibrary/files/opencv-ios/</a>. Alternatively follow this guide <a class="el" href="../../d5/da3/tutorial_ios_install.html">Installation in iOS</a> to compile the framework manually. Once you have the framework, just drag-and-drop into XCode:</p>
<div class="image">
<img src="../../xcode_hello_ios_framework_drag_and_drop.png" alt="xcode_hello_ios_framework_drag_and_drop.png"/>
</div>
<p>Also you have to locate the prefix header that is used for all header files in the project. The file is typically located at "ProjectName/Supporting Files/ProjectName-Prefix.pch". There, you have add an include statement to import the opencv library. However, make sure you include opencv before you include UIKit and Foundation, because else you will get some weird compile errors that some macros like min and max are defined multiple times. For example the prefix header could look like the following: </p><div class="fragment"><div class="line">//</div><div class="line">// Prefix header for all source files of the &#39;VideoFilters&#39; target in the &#39;VideoFilters&#39; project</div><div class="line">//</div><div class="line"></div><div class="line">#import &lt;Availability.h&gt;</div><div class="line"></div><div class="line">#ifndef __IPHONE_4_0</div><div class="line">#warning &quot;This project uses features only available in iOS SDK 4.0 and later.&quot;</div><div class="line">#endif</div><div class="line"></div><div class="line">#ifdef __cplusplus</div><div class="line">#import &lt;opencv2/opencv.hpp&gt;</div><div class="line">#endif</div><div class="line"></div><div class="line">#ifdef __OBJC__</div><div class="line">    #import &lt;UIKit/UIKit.h&gt;</div><div class="line">    #import &lt;Foundation/Foundation.h&gt;</div><div class="line">#endif</div></div><!-- fragment --> <h3>Example video frame processing project</h3>
<h4>User Interface</h4>
<p>First, we create a simple iOS project, for example Single View Application. Then, we create and add an UIImageView and UIButton to start the camera and display the video frames. The storyboard could look like that:</p>
<div class="image">
<img src="../../xcode_hello_ios_viewcontroller_layout.png" alt="xcode_hello_ios_viewcontroller_layout.png"/>
</div>
<p>Make sure to add and connect the IBOutlets and IBActions to the corresponding ViewController: </p><div class="fragment"><div class="line">@interface ViewController : UIViewController</div><div class="line">{</div><div class="line">    IBOutlet UIImageView* imageView;</div><div class="line">    IBOutlet UIButton* button;</div><div class="line">}</div><div class="line"></div><div class="line">- (IBAction)actionStart:(id)sender;</div><div class="line"></div><div class="line">@end</div></div><!-- fragment --> <h4>Adding the Camera</h4>
<p>We add a camera controller to the view controller and initialize it when the view has loaded: </p><div class="fragment"><div class="line">#import &lt;opencv2/videoio/cap_ios.h&gt;</div><div class="line">using namespace cv;</div><div class="line"></div><div class="line"></div><div class="line">@interface ViewController : UIViewController</div><div class="line">{</div><div class="line">    ...</div><div class="line">    CvVideoCamera* videoCamera;</div><div class="line">}</div><div class="line">...</div><div class="line">@property (nonatomic, retain) CvVideoCamera* videoCamera;</div><div class="line"></div><div class="line">@end</div></div><!-- fragment --> <div class="fragment"><div class="line">- (void)viewDidLoad</div><div class="line">{</div><div class="line">    [super viewDidLoad];</div><div class="line">    // Do any additional setup after loading the view, typically from a nib.</div><div class="line"></div><div class="line">    self.videoCamera = [[CvVideoCamera alloc] initWithParentView:imageView];</div><div class="line">    self.videoCamera.defaultAVCaptureDevicePosition = AVCaptureDevicePositionFront;</div><div class="line">    self.videoCamera.defaultAVCaptureSessionPreset = AVCaptureSessionPreset352x288;</div><div class="line">    self.videoCamera.defaultAVCaptureVideoOrientation = AVCaptureVideoOrientationPortrait;</div><div class="line">    self.videoCamera.defaultFPS = 30;</div><div class="line">    self.videoCamera.grayscale = NO;</div><div class="line">}</div></div><!-- fragment --><p> In this case, we initialize the camera and provide the imageView as a target for rendering each frame. <a class="el" href="../../db/da9/interfaceCvVideoCamera.html">CvVideoCamera</a> is basically a wrapper around AVFoundation, so we provide as properties some of the AVFoundation camera options. For example we want to use the front camera, set the video size to 352x288 and a video orientation (the video camera normally outputs in landscape mode, which results in transposed data when you design a portrait application).</p>
<p>The property defaultFPS sets the FPS of the camera. If the processing is less fast than the desired FPS, frames are automatically dropped.</p>
<p>The property grayscale=YES results in a different colorspace, namely "YUV (YpCbCr 4:2:0)", while grayscale=NO will output 32 bit BGRA.</p>
<p>Additionally, we have to manually add framework dependencies of the opencv framework. Finally, you should have at least the following frameworks in your project:</p>
<ul>
<li>opencv2</li>
<li>Accelerate</li>
<li>AssetsLibrary</li>
<li>AVFoundation</li>
<li>CoreGraphics</li>
<li>CoreImage</li>
<li>CoreMedia</li>
<li>CoreVideo</li>
<li>QuartzCore</li>
<li>UIKit</li>
<li><p class="startli">Foundation</p>
<div class="image">
<img src="../../xcode_hello_ios_frameworks_add_dependencies.png" alt="xcode_hello_ios_frameworks_add_dependencies.png"/>
</div>
</li>
</ul>
<h4>Processing frames</h4>
<p>We follow the delegation pattern, which is very common in iOS, to provide access to each camera frame. Basically, the View Controller has to implement the CvVideoCameraDelegate protocol and has to be set as delegate to the video camera: </p><div class="fragment"><div class="line">@interface ViewController : UIViewController&lt;CvVideoCameraDelegate&gt;</div></div><!-- fragment --> <div class="fragment"><div class="line">- (void)viewDidLoad</div><div class="line">{</div><div class="line">    ...</div><div class="line">    self.videoCamera = [[CvVideoCamera alloc] initWithParentView:imageView];</div><div class="line">    self.videoCamera.delegate = self;</div><div class="line">    ...</div><div class="line">}</div></div><!-- fragment --> <div class="fragment"><div class="line">#pragma mark - Protocol CvVideoCameraDelegate</div><div class="line"></div><div class="line">#ifdef __cplusplus</div><div class="line">- (void)processImage:(Mat&amp;)image;</div><div class="line">{</div><div class="line">    // Do some OpenCV stuff with the image</div><div class="line">}</div><div class="line">#endif</div></div><!-- fragment --><p> Note that we are using C++ here (<a class="el" href="../../d3/d63/classcv_1_1Mat.html" title="n-dimensional dense array class ">cv::Mat</a>). Important: You have to rename the view controller's extension .m into .mm, so that the compiler compiles it under the assumption of Objective-C++ (Objective-C and C++ mixed). Then, __cplusplus is defined when the compiler is processing the file for C++ code. Therefore, we put our code within a block where __cplusplus is defined.</p>
<h4>Basic video processing</h4>
<p>From here you can start processing video frames. For example the following snippet color-inverts the image: </p><div class="fragment"><div class="line">- (void)processImage:(Mat&amp;)image;</div><div class="line">{</div><div class="line">    // Do some OpenCV stuff with the image</div><div class="line">    Mat image_copy;</div><div class="line">    cvtColor(image, image_copy, COLOR_BGR2GRAY);</div><div class="line"></div><div class="line">    // invert image</div><div class="line">    bitwise_not(image_copy, image_copy);</div><div class="line"></div><div class="line">    //Convert BGR to BGRA (three channel to four channel)</div><div class="line">    Mat bgr;</div><div class="line">    cvtColor(image_copy, bgr, COLOR_GRAY2BGR);</div><div class="line"></div><div class="line">    cvtColor(bgr, image, COLOR_BGR2BGRA);</div><div class="line">}</div></div><!-- fragment --> <h4>Start!</h4>
<p>Finally, we have to tell the camera to actually start/stop working. The following code will start the camera when you press the button, assuming you connected the UI properly: </p><div class="fragment"><div class="line">#pragma mark - UI Actions</div><div class="line"></div><div class="line">- (IBAction)actionStart:(id)sender;</div><div class="line">{</div><div class="line">    [self.videoCamera start];</div><div class="line">}</div></div><!-- fragment --> <h4>Hints</h4>
<p>Try to avoid costly matrix copy operations as much as you can, especially if you are aiming for real-time. As the image data is passed as reference, work in-place, if possible.</p>
<p>When you are working on grayscale data, turn set grayscale = YES as the YUV colorspace gives you directly access the luminance plane.</p>
<p>The Accelerate framework provides some CPU-accelerated DSP filters, which come handy in your case. </p>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:37 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
